Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Self-adaptive Web crawler code generation method based on webpage source code structure comprehension
Yao LIU, Ru LIU, Yu ZHAI
Journal of Computer Applications    2023, 43 (6): 1779-1784.   DOI: 10.11772/j.issn.1001-9081.2022060929
Abstract322)   HTML20)    PDF (1224KB)(115)       Save

To address the problems of Web crawler code failure and high manual maintenance cost caused by webpage source code changes led by frequent webpage redesigns, especially changes in element structures or attribute identifiers of target entities such as article dates, main body of text or source organizations, a self-adaptive Web crawler code generation method based on webpage source code structure comprehension was proposed. Firstly, the corresponding Web crawler code was extracted by analyzing the change patterns of webpage structural characteristics. Secondly, the changes in the webpage source code and code were represented by the Encoder-Decoder model. By fusing the semantic features of the webpage source code structure, the features of webpage source code changes and the features of webpage code changes, an adaptive code generation model was obtained. Finally, the perception, generation and activation mechanisms of the adaptive system were improved to form a Web crawler system with adaptive processing capability. Compared with TF-IDF+Seq2Seq and TriDNR+Seq2Seq models, the proposed adaptive code generation model was experimentally verified to show the superiority in the representation of webpage source code changes and the effectiveness of code generation with a final accuracy of 78.5%. With the proposed method, the Web crawler code operation problems caused by the webpage source code changes could be solved, and a new idea for the adaptive processing capability of Web resource acquisition — Web crawler technique was provided.

Table and Figures | Reference | Related Articles | Metrics
Automatic focusing algorithm based on improved gray contrast function
HUANG Wei-qiong YOU Lin-ru LIU Shao-jun
Journal of Computer Applications    2011, 31 (11): 3008-3009.   DOI: 10.3724/SP.J.1087.2011.03008
Abstract876)      PDF (457KB)(476)       Save
To meet the fast and accurate requirement of image measurement in the auto-focusing system of image measurement instrument, gray contrast function was improved as following: by making use of the features that the focus image has smaller range of the gray scale transition than the defocused image and calculating the average change in gray value, focusing was reached with the number of changes in gray-scale. The comparison shows that the improved gray scale contrast function has simpler time computational complexity and higher focusing sensitivity. The improved function, with good stability, has faster focusing speed and higher focusing accuracy than other methods in the auto-focusing system of image measurement instrument.
Related Articles | Metrics